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ABSTRACT ' ' ' . 

The Austin independent School District received an 
Elementary and Secondary Education Act Title III grant' in 1973 to> 
develo^p an internal research and ^evaluation capability. Funding was 
provided the resulting Office. of Research and Evaluatign (ORE) for 
three years. The foci of the original grant 'n^re (1) tc. develop a 
district evaluation mod^el, (2) to evaluate federal, and district/ 
developQ|ntal programs/ (3) to coordinate research dpne in the / 
district by outside researchers., and (4) , to develop the Irfexas- Joint 
Orban Evaluation Council to promote ij^f oirmation-sharing among the 
seven larg^ Texas citie^. Of seven objectives for the final year cf 
the pro^ject, six were achieved, including* refinement and / 
documentation df the evaluation model/ planning and ref inemeiit- of tif.e 
instructional program evaluation data* base, and' dissemination of 
project informatitn locally and nationally. An evaluation in/dicates 
that the ptoject should be continued when federal funding, jruns cut, 
though some organizational changes may be xeguired, 'Future goals for 
ORE include impro\iing the technical . quality cf evaluations, [the 
internal efficiency of CRE operations, and the understanding in the 
district of information provided by'-ORE, A history of the c}eat?.on of 
ORB and a descri^ption of its organization and functions he lb document 
the -^evaluation of the, project. (Authpr/PGD) - - I 
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ABSTR>pT- 

- s EVALUATION OF TITLE III EVALUATION PROJECT, 1975-76 

Description of Program / ^ 

The third year of a contihuing' ESEA Title ^III grant was awarded to the Austin 
Independent School District in Septmeb'er,' 1975, to continue the design anii 
|.mplementation>.af^a.model evaluation capability within the district* The 
grant funds totalled $66,650 for the period Se^ember 1, 1975, to June 3P, 1976, 
These funda^pported evaluation personnel ii/2 senior evaluator,« 1 process 
evaluatory^ evaluation interns,' and 1 Secretary), operating expenses for 
AISD evalaation activities (daj^a processing,^of f ice supplies, evaluation 
materials, etc.), and travel expenses fox the Joint Urban Evaluation Couilcil 
of Texas • . ' 



Tbtef original foci of the Title III evaluation grant first awarded to AISD in 
1973 were* w - ^ ^ 



/ 



•I /, , 1) Development of an AISD evaiiiation* model. 

-Evaluation of federal and district developmental programs* 
^ Coordination of research done in the district by persons external 

:j \, to the disjtrict. , ' /.^ 

-4) Development of \a Texas Joint ^Wfban Evaluation Council^ (JUEC) to promote 
information-sBH^ing among the seven large Tex^s cities.- 

, / \ 

. Three years of Title III funrfs continifed and, expanded the work toward these 
objectives (see next section for a listing of the specific 1975-76 'Title III 
objectives),. ^ » ' - 

' f . • . ' " ' ' 

3 Evaluation Purpos 

The major decision 'questions addressed for the third' year by this report 
, /listed below: * , , ' I 



are 



/stem-rL^vel ^ ^ • 

1) Should the project to design and implement a model evaluation 
capability in th^ •Austin Independent School District be cjontinued, 
expanded, or""discontinued? ' ^' ^ J » . .. ' 

2) * Shou^ additional- local 'funds t^B committed to this project effort? 

Program-Level ^ • , 

^ 1) ^What should- be the optimal, internal organization of the evaluation* 
<l\ ^at should, be the project ob^ectiyes for the 191f>-ir school year.? 



The seven specific objeqtrives set for achievement by\ this yearns project are 
listed below: 



1) Complete the refinement and docu^ientation of th^ CIPO 6vaXi>ation tnodel. 
Plan and refine the instructional program evaluation data'^se, 
*3) ' Promote understanding of evaluation informaticfh^and procsduV^ ambng 



< 



AISD staff 

M) In'stltutionalize the AISD-CIPO evaluation model, ^/ > 

* 5) Administer and/oir coordinate evaluation 'activities of the district. 

*6)"-'Disseminat6-^ project information at state anxi national^^evels . 

7) Perform efficiently all other obje'ctives outlined, ^ x ' 

New O.R.E. foci fqr the 1975-76 Title III project are ^marked above- by asterisks. 

The purpose of this final evaluation report is to present information and evi- 
denceNto help answer the above four decision questions and to asses^s whether - 
or not the seven project objectives* for 1975-76 were achieved, 

Anofher purpose of this evaluation report is to document the , development of 
,the Office of Research and Evaluation in AISD, This VeporC therefore includes^: 
a history af the 'events which led to the creation pf the office; a -descjrip- 
tion of_the organizational structure and functions of the office whic^i tiave 
evolved over- the last ihr-ee years; and indications of future goals and direc- 
tions which the office now^aces,, A. reading qf this particular report will 
hopefully .serve as an orientation fbr c5uVrent and new district employees and 
other interested persons who wish to ui^derstand— the° p hil o s op hy ^^fed -impi^ 
tionr o( evaluaf'iort in AISD, * ^ ' ' ' 



Evaluation A ctivities ^ ^ ^* ' » ' ' 
■ I • /* • 

The events reported^ on here are for the most part narrative documentation* by 
0,R,E. staff ofthe" activities thfey iiave* carried out during 1975-76 in order 
'to^a:chieve the 1975-76 Title III..objective*, ,In addition, events" related to 
the development of the Office of Research^arid ^valuation are' rec*orded here as 
they were observed, by current O.R,E; staff members. It ds certain that"^ there 
were many other factors and events operating during the period reported on here^ 
wliich thfe -wrttersr have>ver-looked or have npt emphasized, Therefor^e,. It shojild 
be made clear ta the 'reader tj;i^t thiSL. report is written $trictly from the view- 
point* of O.R,E. ^jersonnei anid should not. be considered an unbiased eyaluabion 
of the Title III iJroject. nor of the elf f ectiveness of the AISD Jff ice of ' Research 
an4 Evaluation. ' ' r . ' , ' , - • 

A separate .evaluation, of this project has been contracted tp Dr, Eloyd Brandt, 
a-protesSor^f management and Director of Planning at The University of Texas , 
at Au6tin,^/arid is now in. progrese. His report will be completed soon and is 
publishedy^s: an appendix.'. • • • . 
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Evaluation Findings ' * . \ ^ 

Because O.R.E. personnel are probably the appropriate ladministra^tive .stj^ff to' 
make recommen'datigns on the four decision questions audressed by this re'pxDrt, 
recoonnendat^ons are implicit . in' the findings reported here. ^ -A 

The progress made toward establishing an -effective and j5roductiv6 AISD evaluati^on 
unit oyer the ^ast three' years indicate that this unit* should indeed bte continued 
in AISD. Evidence for this recommendation includes the number and scope of eval- 
uation projects^ carried o'ut-by O.R.E, and the administratis .and fiscal support 
already given* to the unit. . - ' 



Because the current district resources- aLready allocated to O.R.E^. are adequate, 
to carry out the' office's assigned responsibilities, there is no need for a<^ditional 
^local funds to be committed to O.R.E. at this^ time This finding is augmented by 
the fact that many (j>ther , needs must be met by^ the finite ' ^iaancial resources of' 
the district. 'Should, the of f ice' be Assigned addition^l^tasks for 1976-77 , however, 
there would haye to be commensurate resources allocated for their completion. 

^ Regarding, the internal organization of'^O.R.E. , there appears to b€ sopje need for ^ 
a change f;-om the^c^igin^l structure propos>ed i^n the evaluation model' developed 
/by O.R.E. in 197?-7^; This reorganizatij^on' will require, the creation c5f six sub- 
units within O.R.E. compared to jthe three subunits cohce^Jtualized in the earliet 
model. These six unit's are: ^ Federal Hrograi^ Evalv/ation U'rvit , . Systemwide Evaluation 
^Unit, 'Evaluation Se'rv.ices Unit, Local Project Evaluation Uijit, Evaluation Training 
' and "Dissemination Uni.t, and'External Research Liason Unit, This reorganization re- 
quires |io additional personnel or resources, and possesses the followjing advantages' 
er older structure: The activities of each evaluation project will 



over 



ERIC 



^aah evaluation project will be clpsely 
supervised by a. 'senior Ikv^l staff member who^will have moire time to devote to this 
S task than the C!^.R.E. Coordinatqr would if she "^directly siipfrviled all evaluation 

project activities.. The evaluation activities for- t^^e Fe'deral programs being eval- * 
'uatpd by O.R.E.' ca-a bfe better ccrcjrdinated^.by th^i one senior evaluator who wiM head 
up this .new Fedetal program evaduVtion subunit. Lastly, this ^reorganization will 
allow the evaluation training an^d dissemination activities to be^made'^ more readily 
available to all O.R.E. evaluation projects;^ the increased emphasis on these two \ 
functions ihplicit in the c?reation*of the training and dissemination soibunit will, 
hopefully, increase the effectiveness of O.R. E.\ evaluation projects^ in 1976-7*7. 

The last decision question 'addressed by this report^ concerns the future^goals of 
O.R.E. The general area^ of improvement indicated for 1976-77 incl-ude: (1) im- 
proving «the technical quality of Q.R.E. -conducted evaluations; (2) imjiroving the 
internal ef f ic^ifency^f O^R.E. operations; (3) increasing the in-di§trict under-^ 
standing of O.R'.E. '-obtained evaluation information; (4) pro'vidi-ng* better 'and more 
time.ly input evaluation for p^rogram designV prep»are<4 by the district; knd (5) es- 
tablishing more realistic and relevant evaluation designs, especially timelines. 

Of the seven objectives for the 1^75-^6 Title project, six- were judged as 
achi^ed.per the findings presented in this ^^|^t. ^All 2i Title III activities 
proposed'to be implemented to'acheive these seven objectives were indeed! carried 

' out by Title III ^nd other O.R.E. sta,ff. These are documented in detail in Chapter 
IV. These activities included: evaluating seven m^joir instructicft^al programs; 
assuming the administration of ^nd making many nee'd^d improvements in the' AISD 

..group tes.ting program; providing training in evaluation to Aisb personnel;^ develop- 
ing district evaluation policies and procedures; providing Requested evaluation 
servicesL to AISD. staff ; and making presentations oti O.RtE. aS'tivities at state 
and national levels. v 



..DECISl'Dfl questions" ADDRESSED 



In proper context, the d^ision-^uest?ions' for an -evaluation af^e formulated by 
the decision makers inv9lved, v/ith technical assistance from, the evaluation > 
staff during the design phase of the ^valuation. Evaluation then serves the 
decision-making process by prb^^iding information relevant to those questions' 
and assisting t'^e appropriate administrator's to a;:rive et a recommendation 
concerning th0 decision. Ultimate' responsibility for. making the decisions 
always rests with. the particular decision-makers charged with tTiat re'sponsibif ity. 

Far the 1975-1976 school year a' different procedure has been adopted regarding " 
the recommendations. Formerly', the O.R.E.' staff made recommendations based on 
their perceptions of the evaltiation findings. This year the policy adopted 
in AISD is for O.K. E. to provide the ^relevant ' decision-makers and administrators 
in the district with a copy of the decision questions: and evaluation findings. 
These administrators will have responsibility for making recoimnendations, which 
will be forwardeci to tha Board of Trustees along with the final repo\-t. However, 
since O.R.E. is %n this case the appropriat'e administrative unit to ma^. the 
recommendations, its positions on these questions are somewhat implici^n the 
findings presented, below. * . _ ' ' \ 



A. SYSTEM-LEVEL 



Should the^^roject to design and implement a model evaluation capability* 
in the Austin Independent School Distridt be continued, expanded, or 



discontinued? 



RELEVANT FINDINGS: , ' . ' • / • " 

During 1975^76', O.R.E. was able to make considdrablL progress^ toward 
its programmatic objectives which included: conducting evaluations of 
seven iq^jor instructional progr^s, ^refining the instructional program 
data b^se, jromoting^nderstanding of evaluation information apd ptocedures 
among AISD staff, and institutionalizing the AISD's CIPO evaluation model. 

The capa{)ility to evaluate programs in AISD which has been developed oVer 
'the last three years h^s been utilized to evaluate the major Federal and 
district-funded instruct^nal programs, ^ The needs for accountability and 
renewal are perceived so Sttongly in the district ,that the discontinuation 
of O.R.E. 'activities would result in a real loss to the district,' as 
well as result in a loss of district credibility to the community. 

^RECOMMENDATIONS: " • '■ ' . ' 

The Office of Research and Evaluation' should be continued. 



'J 



2. Should additional local funds be' committed tov this proieot effo'rt? 
RELEVANT FINDINGS; * ^ * • ' 

O.R.E. currently H^s a gqod balance between^ the' resources nee^^ to ac-« 
complish Ov^.E. 's^assignments ajid the amount of resources, which can be, 
V efficiently man/iged at thjis time within the current internal structure 

of the office. The local district's fiscal support of O.R^E.'s efforts' v 
have grox^ considerably over the last three yeara, particularly during 
the 1975-76 school year. ■ 

■ r- 

The* only area where O.R.E. ^s resources are\severely lacking 'is in the 
arfea.of '^input evaluation''. In this type of -evaluation, O.R.E. wousi- 
conduct upon the request of AISO sta.ff-, extensive reviews of current and 
_ past research, identify past and current projects which sought to imple- 
ment treatments of interest to the requestxDr, atld identify other, sources 
of ihformatibn including consultants^ about- the topic(s) requested. 
O.R.E. has -submitted an ESEA Title IV, Part C proposal to TEA this spring 
for -this purpose, but no notification has been received from TEA at 'this 
date regarding its funding 3tatus. ^ ^ ' ' 

RECOMMENDATIONS : * ^ ' , ' " 

Since ^he office's present resources are adequate for carrying out its 
current assign^nents for conducting accountability evaluations, and be- 
cause the other financial responsibilities, of the district are such 
that additional 'allocation of ;resources to 0,.R%E. is noV waf ranted, it 
is probably not wise st this point for' the district to do so.' This re- 
commendation is, of course, -subject to any changes In 'the number and 
scope of O.R^.E.'s current duties and to changes in the current internal - ^ 
jnanagement structure of the office. Should the Title IV application menr- 
tioned .above not b^approved, cutrent staff should incorporate -this area 

e priarity consideration ih feature acMvlMes-; ^ — -~ — =- 

/ • " 

PROGRAM-LEVEL QUESTIONS ^ - " .* . ' ' " ^ 

1. What sht^uld be the optiotl/l internal organization of the evaluation unit? 



RELEVANT FINDINGS: ' . 

For the first three years of its operation, O.R,E. 
on the following theoretical internal structure-: 



has modeled its activities 



O.R,E.« Coordlxuitor 




r 



Figure II-l: Theoretical Internal Structure of O.R.E. During 1973-'76. 



However, the actual managerter)t structure of the office has. b e'en much 
less departmentaiized than is reflected in Figure II-l. Figure li-2 
represents a more true picture of the management struotur^-of the ' 
office for these last three years i 




Figure II-2: Actual ^Internal Management Structure of O.R.^E./liuring ;973-76 

The inoreasing number of evaluation projects assigned ^to O.R.E. ov4r - ' 
the last three years culminated during 1975-76 in an awar^riess tihat 
a mid-level Jf management between the O.R.E. coordinator and at-T.east 
some evaluation projects was .indicated. Since the majority of the ma-ior ' 
projects ev^uated by O.iR.E. (four out of seven) Sre compensatory educa-' 
t±on programs which operate in subsets of a, larger subset of AISD elemen- < 
tary schools, a great deal of coordination must occur among these compen- 
satory evaluation components. These two facts .define the proposed man- ..* 
agement subcomponent' within O.R.E. for federal program evaluation coor- *. 
dination which is^ shown in Pigure II-3 on the following page. 
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The need for the* office to improve Iji-district understanding of , evaluation 
procedures and information continues to be felt. The training and^ dissem- 
ination setvices developed in CT.R.E. this. year tl\rough Title III i^irsources 
**^^lggg^n to. meet this need for se^A^eral evaluation projects during 1975-76, 
but ihis function needs to be extendgdf to all O.R.E. projects. This' 
continuing need indicates a corresponding need for some type of unit. with- 
in O.RkE.^which would coordinate 'evaluation training and dissemination 
for the entire office. 

The office has bnegun to deliver an increasing amount of "evaluation 
services" to district personnel who request assistance in developing 
objectives , analyzing data, pei*forraing small-scale evaluations, etc. 
There are also, unfortunately, many requests, for services which can- 
not b*e filled because of limited O.R*E. resources. As more resources 
do become available to O.R.E^, it sSeffis appropriate that these service ' 
functions be carried out in a coordinated fashion. This currently small 
but growing number of service activities indicates another subunit of* 
O.R.E. which could be supervised at a level below the office coordinator 
position. . 

' ' ' ^ • - , , ' 

The functions of' liasotiing with, external R .and E agencies, i.e., local 
universities, other' large city R and E of ^ices, . contract agencies,* ^ 
professional organizations, etc., are carried out and/or supervised 
^ery closely by .the O.R.E*. Coordinator. Therefore, any revision .In ; 
•O.R.E. .*s internal management" structure should reflect this close rfil^^ion- 
shi^'^'^etween the CojEj^S^lilator and these functions. ^ ' 

^he systemwide evaluation component Already operating in thejciirrent 
offfce organizatio,n has shown 'no need of change. Its. majot* functions 
will continue to be developing ^nd, maintai^it%^aii, instructional data 
base 'for progi?am evaluation, providing bath shojrt and long term looks 
at the total insti^Uctional program of the district,, and providing tes'tr 
ing and other informtion to schools, to teachers, and to individual 
students when it isyappropriate and/or requested ^qt rae insti^uctionaX 
leadership of AISIJ^' . . * • - > 4^ 

The district hasf4i^scted And suppliecj an increasing number of., resources 
for the office to c^arry Sut evaluations of major district-funded programs, 
e.g., the new quartefr. system, local compensator^ education, efforts , etc* ,< 
Although the local programs ^^signe^ to O.R.E, 'tor evaluation in the past 
have been so* diverge as to^ require minimal Coordination among the evaT- 
uation efforts, there. Is perhaps some merit in grouping "these activities 
Under One subunit: l^iis last grouping would complete the internal man- 
agement ' structure of OiR.E. as it appears in Figure 11-4*^ 
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Figure il-4.: Proposed- 1976-77 .Internal Management Str^icture of O.R.E. 

Figure 11^4 represents only a general view of the organi^tional and 
manalgement changes which will be made in O.R.E« 'Th6. details of this 
new structQre will be wp^lced out in th6 late-^summer during a. planning 
retreat attended by senior, level O.R.E. staff. ,.The majot advantages 
of this new structure ate: ' 

(1) The actlvit~l^pf each evaluation project will be closely 

•supervised. by>s^enior level staff member who will have more 
^ . . time to devote to this task than the O.R\E. Coordinator 
would- it she directly^Jsupervised all evaluation ptoject 
'activities. 

-(2) The evaluat;lDn activities for the Federal programs being 
evaluated ,by O.R.E. cafi be better coordiiiatid by the one" 
, senior* evaluator. who- will head up the -new-'Vederal .program 
evaluation subunit. ' ' . . - ~ 



(3) The ^valuation^ training and dissenfl^piiTjn activities can be 
^made more readily available to* all 0»R«E« evaluation projects; 
the increajsed emphasis on these ^two; functions implicit in the 
creation ot the training* and disse^taination subunlt will, hope- 
'fully, increase the effectiveness of these *O.R.JE. activities 
. in 1976^77, ^ ^ .-^ 

REc6M^lENDATI0NS: ' " ^ ' ' ^ ' i* * 

I 

It* is 'rectommendgd ^hat the proposed structure Qutlined^ above be 
iinplemented. This yeat, rec\)iTimen^ations will be made by* the approT 
« prjLate adtaiAistrative staffs * ' ' , • , » . 

2. What §hould be the objectives for the 1976-77 school year? 



J' 



^RECGMMfiNDATIONS : 



Although there will be no Title III evaLuatiori project in O.R.E. 
* next year, O.R.E. will of course continue to set objectives to help 

improve its functioning in AISD. The specific objectives will be 

es-tablished by O.'R.E. senior" level staff during the planning re- 
. trear scheduled for late July, 1976. The anticipated ^areas for 

setting objectives for next year ajre^i 

■ '■ . -Jl.) Improving the technical quality of O.R.E. -conducted evaluations. 

• ■] - ' • ■ 

.2) Imprcxving the internal e;fficiency of, O.R.E. operations.- 

'I' * . 

3) Increasing the in-district understanding of O.R.E. -obtained 

evaluation infprmation. 

4) ^ Providing better and "more timely input evaluation for program 

des:.gns prepared "By the distric't. ' * ^ 

5) Establishing more realistic anS relevant evaluation designs. 



especially timelines. 
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PROJECT DESCRIPTION 



A. •- PROGRAM DESCRIFTION 



QveraU Focus; The Austiri-Indeperident School District's Office'.of Research 
. and Evaluation (O.R.E.) waS awarded kn ESEA Title III grant in Sp&Lng 1975. 

Thi-s wa^ the, third continuing \year that the office hacl received Title III 
^ monies. The original grant, awarded in 1973, was given to« tjie district 
to establish a research and' evaluation capability in the 'district which 
could ov'ersee and conduct .prggraiil evaluations and "coordinate research .activities 
within the district. .Up until 1973, all program evaluations .had beeit conducted ' 
■ by external evaluation, agencies, on a contractual basis. ;'' • . • * 

In 1973, the Austin Research and Evaluation unit; which was created by this 
Htle III grant assumed, the"" following tasks: 

... . • ' • .... 

Development of an AISD 'evdlua.tion model. " ' - . . ■v 

Evaluation of federal an^ district developmental "programs. ' ' 
Coordination- of research done in the -district ty persons external ' * 
to the district. 

Another focus df Austin'^Title III grant^as to' fund in. pkrt the activities of 
■the Texas Joint Urban Evaluatioa Council, composed of- the research directors 
and other staff membe^ qf R and E units in the' seven laafge .^exas urban districts: 
Austin, Corpus Christi, Dallas, El J'aso, Fort-Worth, Hbuston,lana San Antonio. 
A San Antonio Title III grant provided the othex funding for tftls grqup.. 

The two. original purposes of the Council .were (1) to provide 'the two newest 
R and E units (AvStin and San Antonio^with guidance from the more mature 
units in the other five Texas cities, and (2) to facilitate information sharing . 
among. th6- seven cities and promote^ joint problem-solving during the 4-5 meetings 
held each year. ' , ■ " 

Activities 'direcnjed toward the above iajpr objectives, of the Office of Research** 
and Evaluation continued during the ^Irst two years! At the end of the Second 
year of the office'-s ejfistetice" (and- .the end of the'second. year of the office's 
Title • III .granrt) it began to-be cleatv where additional areas of development 
for the office were needed. ^During the first Wyears.of the office's operation, 
the achievement testing program was managed by another department in the disfertct. 
This arrangement Sneant that (5.R.E. had no data quality control over the collecting 
and processing of that information. It gradually be'cdfce obvious that O.R.E. was 
among the. prime users in ' the, district of the testing. irAormation. It followed that 

b:^u' ^^.^he department in^rge of collecting this data, 

fiy this time, data quality and data access had become problems which could no 
longer be ignored. . - . . • ' " 



Another target «area tor, the office by the endTf\the 1974-75 school year was 
the trainiflg of district, personnel to help th^em understand 'the workings and 
philosoj»hy/of -the Office of -I^esearch and Evaluation. It had begun to be clear 



wW^h'^i'!''?'/'^fr"^^ would -never make de^^is^Lqns based on evaluation info'rmatio 
which was dei;ived by a proces's 'that they did not understand. . . , 

• ' -.^ ■ •' , ■ ' 

tMnf..-"^ ^f/eed was for O.R.E. to improve the .communicability k its' eval- 
uation reports We were recognizing that all district personnel iimply did 

^stLn^^f ^^^"g "^^^ available through tL rather 

'Standard report format adoptedin 1973-74. ■■ v ' 

» ; ' • • . . 

^wMci'';hf'??r T/°, finy year'of, the Title -III seed grant^under ' 

mMon^hn f ''^'/"''^"'^ seemecf^h^ appropriate time to. disseminate infor 
mation about the project>nd about th^ office at state and national levels. ' 

tS2^vefrT?Mf??^ °' concerns, the" objectives for 'the o'ff ice's 

Z lilT^l J • Established. The nej. areas to b.e worked on " 

in 1975-76 are marked below by asterisks: - - ' 

1. Complete the refinement' and documentation of the CIPO evaluation 



model. 

■*2. Plan and refine. the instructional program hata base. 

•*3. Promote understanding of evaluation information and procedures ' ^ 
among AISD staff . ' < pa 

*4. ■ Institutionalize|-^he AISD-CIPO evaluation' model. . 



5. Administer apd/or coordinate evaluation activities of .the district. 
^ • • • . • • - 

*6. Disseminate projecf information at state and national' levels . 

7. ' Perform efficiently all other objectives outlined. 

||nd|Bpe^: The project was funded for a ten month perip4 for a-total of 
?t)b,650. The project employed„the following personnel: " 

senior evaluator - 

. ^ ■■- ■ ■ ■ 

1 process evaluator 

3 interns . ^ 

1/2 VQE typist clerk , , " . • 

1 secretary ' ... 

• • . ■ ■•. .. • ■ , 



B. COl^TEXT DESCRIPTION " ^ . . \ 

Background of the dfevelapment of O.R.E. ' * • 

% ' \. , » * ^ ^ ^ 

In the early VO's, Austiit nidepehdent School Disb^:ict had launched into many^ ' , 
liew programs, changing the direction of the dlstrict^ward more individualized 
.instruction, more emphasis on affective activities^ and areas, and introducing acr- 
tivities aimed toward easing the negative impact of desegregation. A^ter 1970 
the^district had sought and accepted many .more Federal monies. 

The) new programs jthat had been Implemented ia the district wei-e: Communications 
Skills (a joint project between Title I and Model Cities) in four East Austin' 
schools. Individually Guided Education in eleven elementary schools, high school 
busing. Emergency ^School Assistance Program (which later became ESAA) , programs 
sponsoring the worJc.of Student Community ^Liason representatives, the Human Rela-^ 
tions Councils on each AISD campus, summer workshops in human relations for admin— 
istrato?:s in the areafe of racial attitudes, etc. By 1973, all these changes had-^, >. 
begun to cause majbt concerns In the district among some parents and staff members 
about the direction that the district was taking knd about the effects of those 
changes. * ' « , * * 

* , ' ' ' » 

A push fpT ''jhore basic educatioij'' by several groups of. parents appeared at the . ^ 
end ^^gl973-74 and rose to ±t$ p^eak'by'the end of^the 1974-75 school year. 



\ 



The original impetus tor the creation of. an AISD evaluation unit came from a 
climate created by cit^zen^ who appeared before the School Board demanding 
accountability of the new programs. One Board member responded, "Let's don't 
^'put ai^ymore programs in .tne district unle^s^we evaluate them.\,^^^All this led ' 
I the district ^,to Hegin thinking ♦about devel^ing an evaluatign capability within * 
the district..^-? 

Iti the fall of 1972, the district conducted inteiyviews for a Coordinator ^of * 
Evaluation. " .The Office 'of Evaluation which was then*" envisioned was conceptuatLzed 
a^ being Similar to the, current Office of Staff Development. This office was seeri as 
operating ^n a service mode, rathe^ than in an accountability mo'de. The Coor-', 
dinator would perform some evaluajCion projects and would also function as a 
coordinator of all other evaluation done in" the district, nqt^all of which tfor 
eveu the majority of *w,hich'wouid be' stlpervised or done by that office. . 

Wh^ *^ the- interviews for this new job were concluded,^ it was decided to bff6r 
the job to a specific individual outside the district. This person, however, 
declined the job offer, and the creation of an Office of Evaluation and- the 
corlresponding Evaluation Coordinator position ^were shelved for the time being. * 

In the meantime,\ the disttict contracted two outside evaluations that year 
(1972-73): "'^D >kn evaluation of IGE ($1200), and (2) an evaluatidn of the , 
Communication ^SKills project (apprbxiroa'tely $26', 000). • ^ 

In the^^sprinjg^ of 1973, thevTex;as Education Agency announ^d the availability 
of a very^J-^ge ESEA Title III fund source available to local school districts 
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through proposals foi* new or innovative programs. A priority, area in which . 
those funds were to be allocated was that of accountability. AISD wifote' 
and submitted a Title III proposal which was designed to establish an Office of 
Evaluation in AISD. Austin Independent School Dist,rict personnel,' particularly 
Mr. Marshel A^shley (then Director of the Department of Educational* Development) 
lobbied at TEA for it. Dr. Vance Littleton, AISD Assistant Superintendent of 
Curriculum and Instruction, obtained 'the support of the Chief instructional 
Officers for this project. The district gave this Title III proposal its • 
highest priority ranking among all the Title m proposals it submitted in the 
spring of 1973.. t • . ' / ' 

At- this time public pressure for accouiit^lnlity waS/Tuqunting all over the state. 
Accountability was a'^top^TEA and St4te^ard of Education priority. Certain . 
, TEi\ personnel, particularly Dr. Walter Howard, had worked to have some Title III 
mgnies allocated for-this p^-iority. Also'at this time ,San ^Antonio submitted a 
Title III proposal to fund the developm'ent.of an evaluation model a consulting 
firm and the sub&equent creation, of an evaluation unit in the Sai\' Antoijlb 
Independent School District. Individuals among the Chief Instructional Officers, 
the urban superintehdents, and staff at TEA conceived the idefa of a^^as Joint Urbai 
Evaluation Council (JUEC) as a way to strengthen both Austin's and San Antonio's 
Title III propQsals. These individuals saw the purpose of the Evaluation Couiv<^il . 
as that of sha^-ing evalt^atibn information among, the seven cities, thereby -increasirJg 
the pSyoff 'throughout the state of aU the urban cities' research efforts. Con-r. 
se^uently, both Austin's'and San Antonio's Title III proposals which were submitted' 
to» TEA i^ncluded funding of the JUEC. ^ 

^f'ter AISD's proposal was submitted to TEA, indications w^e favorable that 
the 'district probably would receive the needed Title III monies to start an ^ 
evaluation unit. During this time strong evaluation components were also written 
into other AISD fecJferal proposals: ESAA Pilot, Esaa Biliijgual/Bicultural, and 

' ESM Basic. • . 

In the late sprang of '73, Austin receiveci notice from TEA that their Title 
III proposal had been funded. (San Antonio't; was also fuftdedj In addition, 
^li;ii^ree of t?he district's ESAA prpposals^Vere funded^ two^ of i^hich had re- ^ 
tained their strong evaluation components (Pilot and Bilingual). Thus, in. 
early July, 1973, the Office of Research and Evaluation was born4 

Related Events following in 1973-74 and 1974-75 have .been documented in evaluations 
of O.R.E. priepared^ in past years which are available'in O.R.E. files. 

* ' - ' , ' * 

1975-76 Events Which Affected the Operation of the Title III Project . ^ ' 

^ ^ ' ~ ^ ^ ; ^^^^ 

During 1975-76 th^re were many ih-district events which affected the operation 
of the Title III evaluaition pirojeVt. • - . 

Basic Ski lls Committ ee Activities: ) During, the, fall of 1975, a Basic Skills 
Committee^was appointed by the Schfaol BoarS for the purpose of studying the. 
status af basic skills^ in AusL^in^ndependent School District^and making quarterly 
repbrts to the Board ort this suEject. This. committej^ Requested from* the admin- 
istration a school by school breakdown of air 1974-75 achievement test results. 
Because O.H.E. ha4 fcy^that time assumed^ responsibility for the achievement 
testing program, the responsibility of providing this information fell to O.R.E. 



•It wad important, we felt, to ^provide this group and the community with something 
more than just a .rank order listing of school lev^el grad^ equivalent scores." 
Therefore, the O.R.--E, testing personnel and the Title Ili staff Klevoted^practi- 
, eall-y all of November and December to developing and •producing school piwfiles 
•.wht^ch included .no^vgniy achievement, test data-but attendance, socio-economic, 
stjaffing, and other school level data. (An example of this shown in Figure I'V-l 

»The major point made^here is that this unanticipated' task of producing 
- school by school- profiles early in 'Qie year deti^acted seriously^^rom the 
prpo^ct resources, thereby depriving^the project of two months which had been 
planned to^^be devoted -to the inserVice activities ojutlined in the project pro- ' 
posal.;. , ■ . ^ ■ \ . 

^rricular Context of the District s In responsdv to communitfy pressure ^or ' 
more .basic education", the Administration designated 1975-76 as a year of - 
reemphasis. on basic sJcills. This "reemphasis" was somei^hat of a compromise 
response to^several groups in the community which had requested that Austifi 
^•Independent School District establish some alternative fundamental schools. 

Most of .the federal curriculum programs operating in the di&trict continued 
during -1975-76: Title VII Bilingual, SSAA Pilot, ESAA Basic, apd several 
smaller Title HI projects. . ■ ', 

, * • 

The largest curriculum innovation introduced' intO/ the district this year ' ■ 
was- a^ the high school level. The» state-mandated quarter system was in- .' 
tro^ced into the enti,r^ curriculum Krl2 for record-keeping purposes. How- 
ever, at the high school level ^the new quarter system was' accompanied by a 
massl^ve curriculum revision whi-ch greatly .increased the numter and variety of 
courses offered to high* school sttidents. O.R.E. conducted an evaluation of 
this effort. ^ - . • 

Personnel Changes in t he District ! Just prior to the start of fhis :^ear's 
Title III evaluation project. Deputy Superintendent 'Dr. Vance Littleton 
to whom the office -had reported directly in the past, resigned and moved 
to a superinteh^ency in south Texas. This meant that for the' rest of thi^ 
year O.R^E. has Reported directly fo the Superintendent. 

^ . ' ' 

1975-76 was a School Board election year, 'iwo bo^d members, Jetry Nugent 
and Will Davis, ran for reelection and. both were "reelected. It might be 
noted here that the 1975-76^hool elections demonstrated a. somewhat conser- 
vative feeling among the s^KSfe board election voters. This A in <:ontrast 
to the usually liberal vot& among the Austin citizenrwin city\nd state 
elections. ^ ^ ' • , • , ' 

C. EVALUATION DESCRIPTION ' ' \ 

The evaluation of .the 1975-76 Title HI evaluation' project was designed to , 
be contracted 'to an outside consultant. The project dii^ector had anticiT)ated 
calling in one. or more national experts in public school evaluation, to con- 
dtict an audit of the achievement of all the projett objectives giving special 
^i'^?To^?''° objective--, 'lO.^.E.-wiU promote the institutionalization of 
the AISD's CIPO evaluation model". 
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In early April, Dr.; Jim Jacob^who developed an(d currently directs an ou^-* 
standing research^ and evaluation unit in 'the Cleveland. Public Schools was 
invited to- conduct this evaluation. However, he was siated to dirept a three 
morith project in' Baghdad, Iraq, du^ng May, yune-', and July of 19/6, and there- 
fore was unable to come to Austin. In'May, and early June ,0,R.E, invited 'by 
phone three more nationally-ifecogpized evaluation leaderia but ^11 were unable . 
to schedule time aVay #t6m their current , pro jerts to, perforp the needed ' 0,R,E, 
audit: Drs. James Popham (UCI4) , Dan Stufflebeftn (Western -Michigan) , and Arnold 
Ashburn (Texas Department -of Public WelfaV?). ^> ' . • 

* • » » * * * "* 

At this point", the project director decided to search. in other circles for 
a qualified .consultant" to -do the audit, Dr, Floyd Btan^t, University of* ^ 
Texas prcrfessor of management, -also Director of the* University of Texas' 
orfice of Planning and^ Assistant to President Lorene Rogers, was-^ interviewed 
and contracted to evaluate the project at a cost of $750,60, Dr, Brandt's 
audit is in ,pr6gress /at this date, and his -report will be submitted to the > 
Texas Education Agency under separate cover. ^ \ ^ ' ^ p 



\ * 
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EVALUATION FIND! 
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This yftar. /AISD's Office of Resea^h^ ana'%yaltMion Title III 'p^^*e>lect had sev.en 
objectives. Eacl^ of these obj^ctfves will^be presented iJi^theCfoSbwing pages 
-along- <?ith a statement concerning, the leVel of attaiujnent of that obj'eetlve. - 
Ln.r'nr;!^' designed -to lead to< the achievement of. each objective are^der- 
obj ti i ""^•'^ cprijesporiding Supportive Data section for ea£- 




.. (OBJECTIVE) O.R.E. will complete the r^fi^nement and documeiitation of-tflfe 
CIPO evaluation mo^el. ■ ' "^p • • * .. , " 

V .IT 

LEVEL OF ATTAINMENT:". Achieved.* ^ • 

- • , S " ' » ' ■ . ' ■ ' ' ,v, 

*Although this is stated here as. havii\g.been; achieved,*' it mifet be made clear V* 

that such an objective will probably never ^e atft^k^ned in an absolute sehse. - ' 

Any model, should be designed in sucfh ^ way tha;; cpntincms improvement is • 

internally mandated. Refifhemewts will klways^iave to be made Iq' the CIPO 

planning and evaluation model.. However all/ of t& activities proposed 

in th^ Title III proposal to effect the ^chievfimeA^^of this objective have 

been carried out. ° ' ? 

SUPPORTIVE DATA: ' ' ' " o - ' r*" 

O^R.E..will review draf ts desJgfts with^appropr^ate de'cision mjtjiers (t eachers, 
to admini strators) and revise these desigfis as nece^arv ., ' 



Air seven O..R.E. -developed evaluation , designs fo^ 1975-76 (see <:olumn 2, 
Table IV-1) were reviewed with %ome jppropriaite- decision , (inkers. In some . 
cases, relatively few changes were made in e'valuatd.on,;deg4khs »if ter t*iese 
. reviews^ Signoffs by th^e^ecisdonrmtkers wete requ^'^dired by O.R.E;. to docu- 
ment that they had reviewSd ^d approved the^ Implementation 'of that particular 
evaluation design. ' - • " -• . ^A*^ *» • 

• Problems with the evaluation design revi^eH. nrocP.ss >y"..,r.i nofed that ~ 
getting .the appropriate decision-makers. to reviewjO-rR^E. 's evaluation designs 
continues to be a time-consuming, 4jnglamorous. 1&slt» which" o'f fen is performed 
in a perfunctory manner by) administrative peifi^nel.. ' It 'is ptobable' that 
many of the declslon-make/s , even those^who Vl|ned .off. on -ia^jdesign, did not 
read the desigri befqre dtoing so, .\^.' . i . . •. " . * 

'Perhaps the source of this problem lies Isi .t^he:. Yorm in which: ttie evaluation 
design is presented, i.e., the evaluation desig^Td^cument is. too detailed, • 
too thick,, demands too much time of already OT^ertaxed administrators. (One 
of the continuing sources of noncommunication' between evaluation and program 
personnel- is 'the cdmplexitj: of the evalu^tidn proc'ess'. ).' ' ^ ^ ^ 
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TABLE %rl: SUMMARY OF EVALUATION REPORTS 'PRODUCED BY O.R.E. DURING 1975-76 
. (fioea not include occasional papers, t>rochures 



Evaluation 
Prbject 



, instruments, etc.) 

Technical 
Report ' 



1. Quarter System 

2. Title I V- 

3. State Compensatory 

4. ESAA Pilot 
(Project Assist) 

5. Title I Migrant 



6i A Regressfion Analysis , 
of Pupil/TeacHer R^tip 



Data Services; External' 
User and Internal Staff 
Satisfaction 




8. Sy^emwide Evaluation 

9. Title VtXv Bilingual 



/ 10.' Title III" Evaltj^tion 
. Project 

'-11. ABSTRACTS of 1975-76 
Evaluation Projects 



Totdl 




'^S'^forpaative reports and 67 forgialfive 'memos 



*^ 12^f<jrtnative reports and 67 formative memos 



ERIC. 




O.R.fi. will continue to work on improving communication between evaluation and 
project personnel in the early stages of every evaluation project. It %ay 
wdll be that future O.R.E. evaluation design documents will be very abbrevia^ted 
•summaries of 1) decision >tterdtions, and 2) testing schedules. 

O.R.E. vill execute design plans . / 
• • ^ > ' 7' ^ ' ^ ~ - . 

\. The seven evaluation designs were exe'cated .^urlng 197^-76: 

Quarter System Evaluation . , . * 
Title I Evaluation ' . , * ' 
State Compensatory Education Evaluation 
•ESAA Pilot (Ptojfect Assist) Evaluation' * ' A , 

-Tltle;i Migrant S^^aluatlon . ' , ' 

System Wide Evaluabi^n 
Title VII Evaluation 
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O.R.E. will pr epai^e and prafeent formative evaluation reports . 

During 1975-7^, 12 formative reports Wfere prepared and released by the O.R.E. 
(see Column 3, Table IV-1)."" One O.R.E. evaluation prqject (ESEA Title VII i 
Bilingual Project;) began to experiment daring tlfe year by sending "form^tiveX 
S^os to project staff, rather 'than producing formative repoirts. (During the 
yjar,- 67 such memos wete sent). The object of tfiis' new approac'h is to briak 
/the. evaluation information down into more confrontable smalier'-pieces of in- 
^ormati-on'^d to achieve' faster turnaround on eval4iation\ikormation fed to' 
. program sta^f. lA -the Title VII Evaluation project, j:h'is formative memo 
approach seems to have been more successfuf than the -foma,tive report approach. 
This approach will probably' be , utilized heavily by ojther evaluation projects 
in the future. . . 

^ O.R.E. w ill- prepare and -present summative" evaluation reports . 

, 

This year, Qleyen summatlvfe or final eyaluatlon reports- will be released by ' 
0.R;E. They are: ' . 

1975-76 Final Evaluation Repyts , , ' * 

Systemwlde Testing . • ' - * 

Quarter System ^ . , 

Title VII Blllngual/Blcultural Project . ' - ^ . 

Title I Project , . ' 

State- Compensatory *Educ^tlon Project 

ESAA Pilot Project Assist ' ' " * 

Title I Migrant Project ^ ^ ' ' * , 

A Regress-ion Ana;Lysls of Puf>il/Teacher- Ratio ' \ . ' 

Data Services:- External User and Internal Staff Satisfaction \ 
Tit Id III Evaluation Report* ' . ' - . \ ^ 

ABSTRACTS of O.R.E. Evaluation Final Refloats ^ \ ^ 



... *. .V - . • 

These reports are planned for completion by July 1, 1976 and will then be sub- . 
mitted to AISD administrative staff for review and study. During July, AISD 
administrative st^ff will make r^om^endations regarding the decision questions 
addres^^d by these* reports . Then in early August/ these reports along with 
the administrative recommendations will be presented to the AISD School Board • 

In the previ6u^-£wo years, O.R.t, has made these recommendations, , However^, 
this year the recommendations wllVbe made by administrative personnel^ based 
on the evaluation findings reported Tn these studies. This change in procedure 
was^ suggested 'by AISD's Cabfi>et {Superintendent, Assistant Superintendent, and 
Directors of Er^mentary and^Secondary Education) • ^ . ' ^ 

O.R.E.. will revise the OIPO model and XlSD Evaluation Guide as necessary . 

During the 1975-76 schobl year, O.R>E. developed -a Proposed Initial Set of 
Policy/Procedure Statements Covering Research and Eviiuatioil in^ SD. These * / 
proposed policy s^tatements w6re presented! to the Cabinet for their review and I 
input in late sprung, 1976. The next^step involved in' having these policies 
legitimatized is' for the Superintendent tx^- present them to the School' Board 
for fheir approval. It is anticipated that this event will occur in the fall 
of 1976. Onc^ these policies have^been approved dnd corresponding 'procedures ' 
have been developed, they will be added to the AISD Evalu^ion Model, document 
under the Policies section and will also be inserted in the Admi^iistrative Handbook. 

^ There have been few written changes in the AISD Evaluation Model this year.. / . 

There 'is, one area in the model, however, which will eventually probably be 
changed : that is a change in emphasis from ob^j actives to decision question . * 
Ifw^uld take a long treatise,, indeed, to fully relate the practical ^and ' ^ 

philosophical background surrounding the conflict: in ourj thinking between 
these two focuses.'' However, it can be summarized briefly. Evaluations 
which focus on program* objectives alone wind^ being didactic, fragmented ef- . 
forts which which lose a central focus or^eme.^ Further, an evaluation report 
which reports on progress^ (or lack of progress) of a program toward the achieve- 
ment of forty outcome, process, and input ' objectives , does not e'asily encourage 
renewal or acciguntability in an^rganlzation which,^ changes itself only with/ 
great difficulty anyway. However, evaluations wHich focrus on the 'fewer program de- 
cisions which ^must be made at spedific points in the project's operation by various 

•levels of decision-makers are much. more likely to. result in the evaluation information 
actually being used by decisiori mal^fers. "This is especially true whan the de- 
cision questions have /been generated b^^the decision makers themselves. 

Because the Austin definition of evaluation is' "providing information for de- 
cision making"^, the trend in evaluation .focus has been as^fo^lows: . . ^ - . 

^ 1973-75 Focus mainly on objectives. ^ , 

1975-76 Focus on both objectives and decision' questions. ' \ 
^/ ' 1976-77 Focus mainly on decision -questions. 

0 • ■ ^ , ^ \ 
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» (OBJECTIVE)' O.R.E. will plan and refine the instructional "program data base., 
.LEVEL OF ATTAINMENT: Achieved^ ' , , • -C , 

All evidence indi^c^tes that great strides' have^ been made itv the "directi^ 
indicated by this objective. Again, it^is doubtful that this objectit^e 
cm e^er be attained in^ any, absolute sense, since continuous inrproyements must 
, be mir sued ^However, relative fq the past two years this objective must 'r 
deflnitely^e ascribed as- achieved'. f • ' * , . ' 

SUPPORTIVE DATA:- > ■ - , ^ ', . 



O.R.E^ Assumed the AISD Gifoup Testlrtg ^Program , • . 

During the 1975-76Vchool year, the Office of Re'search .and" Evaluation ^^umeld 
the responsibilitie^E of the group testing program in the district! For the ^' 
pg,st fi<^ /ears, the group, |:^^ing program had been adiriinistef^d b^he Depart- ' 
^nt of^ Student g^velopment^ ' ' ^ ' . ' , • 

Tlje assumption of tjjese responsibilities meant that O.R.E*. supervised the ^ 
following test administrations with the following populations r • • ^ 

TABLE ^2: AISD GROUP ?rESTII<fc PROGRAM. -1^5-76 (j** '* * 



TEST 



POrPULATK 



PAL (Language DcQpinance Test) 
♦ 

Metropolitan Readiness Test 
Boehm Test- of Basic Concepts 

Primary Mental .Abilities . 

Boehm Test of Basic Concepts^ 

California Achievemej>t^Tests 
(Readin'g and Math Subtests) 

Sequential Tests of E^^ 

tional Progress {Reading, 
English Expression., .Science, 
Math,' Social Studies.) 

> ^ 

•School- Sentiment InJex* 



DATES 



All K-2 s'tudent^no? al- 
- readZ-iclassifled as, to 
dominant language 



All firs 



•St gra^ s 



stu4^nts 



All kinder£|arten* students 
in Tpitfli schools 



AM third grade students 

All kindergarten students 

All studenfs^xKT grades, 1-6 
Allf students 4n< grades 7-8 ' 

All 'students iii grades 9-12 



6th grade - sample of 
' classfes ' ^ . - 
g^ade - sample* of 
classes ^ 



Aug -2?-Oct,31^ "75 



'Septem lW-^l.-5. '75 

ft 

Sept 2-5, '75 

Thfoughout^ year 

February 9-12; ^76 

April 5-15, V6 \ 
February 8-13,. '76 

April 12-23, ''76 



April -^gW,^' 76. 



■ 

*This year was the first time tjiis test^was given^ on -districtwide sampling basis\. 
Thidf was -the district's first att^empt to collect infoinnation districtwide Id the! 
'affective area. ' ^ * 
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* Associated Duties ^Assumed by O.R.E. 

1. The testing program doubled In '75-76. The AISD group testing program was 
expanded In 1975-76 to about double Its previous scope. This year, all' students 

all grades Cl~12) were tested with an achievement test. In years past, only 
students In grades 1, 2, 4, 6, 7, 8, ^nd some 9th graders-had been tested. 

2- 0>R>E. scored primary students' tests, prior to this' year, tests for students 
. below^grade A^had been handscored by classroom teachers, and only the result^ had 

been reported by teachers to the district testing officials. This year o.R.E. 
assumed responsibility for hand-scoring all primary students' tests (grades 1-3). 
The pui^poses of this added responsibility were (l)^to relieve overbuifdened teachers 
of this responsibility, and (2) to ensure accuracy of testing rBsults at these 
grades by eliminating errors due to teacher fatlgu:e, misunderstanding of scoring 
directions, etc. ^ • ' 

3- Publication of school by school and district achievement profiles . During 
the e^ly part of , this school year there was a good deal of public pressure 

from both the -3l6cal media, from individual lay groups, and from a Board-appointed 
Basic Skills Committee for. the district ^to release school by school achievement 
test results to the public. This had not been done in Austin for a number of 
years. Therefore, in November and December of 1975, O.R.E. staff developed school 
by school achievement profiles showing the trend for each^ group over the last 
three years. An example of these profiles is shown in Figure IV-1. 

The statistics used in these profiles were: the median, the first quartile 
point, and the third quartile point. Additional information about each group 
was also provided on socioeconomic status of the student populatioii, special 
^ projects operating in that school; student attendance, etc. 

O.R.E. Revised Test Administration Procedures To Improve Data Quality . 

!• Districtwide test guidelines developed . The f irst' ta^^^Wich had to be 

completed was getting' each school in the district to agree on some very crucial 
aspects of the testing program, e.g. deciding what tyt)es.of children would be ^ 
excused from the 'testing program and, conversely, which children must be in- 
' "eluded in the testing sample. 

This issue" and many others just as critical were settled by O.R.E. staff work--s^, 
ing with Testing Committee^. There were four such committees formed to work 
with O.R.E. staff: a committee at elementary, at sixth grade, -at junior high, 
and at high, school levels. (Each of these levels had unique testing situations 
to be dealt with.) . • ' ^ - 

Each committee was composed of people who^ould be working directly with students 
during the testing times. At the elementary level, teachers, principals, and 
counselors were asked to serve on the committee. At the sixth grade, junior high, 
and hl^h school ^evel, mainly counselors were^ involved. In.most cases membership' 
in each committee was based on^a random selection process. At the- elementary 
level!, for example, the AISD faculty directory was used to select most members., ' 
A ^ 

■'■W^tkins, Jim, et., al. , AISD School District and Schpbl Campus Longitudinal 
^ Trends— 1972-73, 1973-74, 1974-75 . Office of Researfch and Evaluation. Austin 
Independent School District, Austin Texas. December, 1975 ' / 
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. "IPigure ly-i: AN EXAMPLE' OF SCHOOL PROFILES PREPARED BY* O.R.E. STAFF IN 
NOVEMBER ANDl DECEMBER, 1975. ■ . ' ■ 
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^It shbikd be reported here that each -of these coiranittees worked long and hai^ 
hours hashing out some very complicated issues like testing dates, what help 
students ^should (or should not) be given, how to give makeup jtest^, how to share 
stud/snts' scores with parents, etc. During the many meetings of these committees 
it was oft^6n heard, "Ther.e are ten different ways to do the testing. ^ Five ways . " 
are O.K., and five ways are not O.K. Our job is to all agree on which one of 
the five good ways we ar^ all going to use." ' , * 

2- Testing guidelines provided . Following the adoption of distrietwide testipg 
guidelines by the Testing Coiomittees, O.R.E. developed detailed testing instruc- 
tions for classroom teachers and other testers in the schools to make^use of 
during the testing situations. These instructions were designed to make them 
appear interesting, lucid, and attractive to school personnel, most of whom 
were not very excited about the subject of standardized testing. 

At the elementary level, teachers ^received very detailed directions as vwell 
as the Testing Manual for their level of the California Achievement Tests. 

The junior high level was the area where *the mpst standardized .conditions had . 
prevailed in years past. Counselors had, in general, already organized the 
testing in their buildings so that it occurred under relatively unstrained 
circumstances. However, a copy of the distrietwide testing guidelines were provided 
for every junior high teacher in the district. ^ * " , ' 

* ^ " 

Testing instructions at the high school level were tite most extensive ones 
provided by^O.R.E. This was necessary here because standa^rdized testing 
was a brand new experience for most of the high school teachers. /The high 
school testing was also the longest, most comprehensive testing which oc-. 
cured, in the district. Instructions for both teachers and building test 
coordinators were color-coded for each day's activities # 

At all three levels, copiojus usage was made of checklists ^or school personnel, 
to follow, checking off dach activity as it was completed. ^6 

3. q.R.E. developed practice tests .for students' use . One of the biggest pro- 
blems which was uncovered during the wdrk with the Testing Coiranittees was that ^: 
many elementary student's (and teachers^) did not feel comfortable with standard- 
ized testing. Much of this discomfort was ascribed to unf amiliarity with tKis ' 
•kind of a testing situation. In response to this problem, O.R.E. developed 
practice tests for use at the elementary level. Practice tests, which resembled 
in format the four different leyels of the California Achievement Tests, were 
developed by O.-R.E. and disseminated for use ty AISD teachers. 

^- Massive s taff dq!velopment conducted . O.R.E. staff (Title III staff mainly), 
opher district administrative personnel (instructional coordinators notably), 
and a core of teachers from thfe Elementary Testing Committee worked long and 
h4rd to/contact every elementary teacher in the district before the testing 
occurrence. These persons presented ^teachers with staff development sessions 
designed to teach teachers how to get their students ready for standardized 
testing, PoiTits were* stressed like: have your students use^number t^wo pencils; 
usd practice tests before the real test to help them be relaxed; have students 
do ^ome of their seatwork' prior to the' testing situation in a\imed situation; ' 
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assign some "multiple-cho;Lce"<^homework prior to the testiAg, A standard in- 
service package (script with transparencies) was developed by 0,R,E, Title III 
personnel. This package was used by O.R.E. staff, central office administrators^, 
principals, and interested teachers tb train ^elementary teachers in these topics'. 

At the junior high and high school leveb.^ school counselors carried out the 
needed teacher training yith minor assistance from O.R.E. staff. 

O.R.E. Planned for Additional Systemwide Data Needs, Especially in Affective 
Areas Related to Instructional Program Evaluation . , ^ 

The.une:^p6cted number of tasksf assox:iated vith a&suming the AISD group testing 
program made it unfeasible to Mo as much in this area as bad been hoped at the 
beginning of the >ear. O.H.E.' was able to collect information related to elemen- 
tary students' attitude toward school and school activities at two grade levels. 
However, the q^iestion of the desirability of utilizing a self-concept test was 
investigated. This investigation resulted in a decision that a self-concept 
test could not be recommended. In a parallel investigation, xB was decided 
collect, on a sample basis, information on attitude toward school. 

At the sixth grade level, classrooms were randomly selected for testing with 
the Intermediate Level of the ^School Sentiment Index (SSI)l. At the' fourth 
grade, ten schools^ were Fandomrly selected, ^n'd all fourth graders. each of 
those schools were tested with^ the Pritaary Level of the School Sentiment Index. 

The data available frpm these 'instruments will provide a benchmark regarding 
AISD student attitude toward school at these grade levels in the years to come. 
Correlations between students' SSl responses and other student data can also 
be made. ' . , ^ ' . . 



(OBJECTIVE) O.R.E. will , promote undeptanding of evaluation information and 
procedures amopg,AISD staff. 

LEVEL OF ATTAINMENT: Achieved. , ' ■ 

» 

All ^activities designed to lead toward the-^ achievement of this ob jective^ave 
been carried out, and there is definitely more understanding within *th€^ diStritt 
ofywhat O.R*E. 's function is, as well as ii;icreased interest in p.R.E. evaluation 
finding^. However;, there is much room for further improvement in this area. 

SUPPORTIVE DATA: . ' ^ - 

O.R.g. will consider alternative approaches to evaluation inservice in a system 
already saturated with inser^^ice activities . . * ~ 

Just prior to the bieginning of ^he 1975-76 project year\ the fueling of the 
district's administrators and teachers was, perceived as, "All our time is, 
taken up with meetings and shuffling papers. ^ There Vs no time to get any. work 

ISchool Sentiment Index (SSI) Both the Intermediate and Primary Level' forms / 
are available from the Instructional Objectives Exchange (lOX) , P.O. Box G, ' ' 
Reseda, California 9133S 



done",- i.e., teach, plan, think. Although perhaps not true in any absolute sens§, 
thdre was a great deal of truth in that expression of frustration. O.R.E.'s task ' 
of communicating evaluation information and the not uncomplesj means by which that 
information is connected could have been seen by many AISD pers?>naiel^ as "just one 
more unnecessary meeting" and "more paper to read". Needless t-o^ say, the task of 
identifying successful 'alternative approaches to evaluation inservice was a chal- . 
lenging one. Gradually, however^ several ideas toojc shape ♦ 

Traditional inservice sessions vere unavoidable for some target 'populations and 
a few of these were conducted'. ' . . . , 

Table* lAT-S below lists those groups who received traditional inservice sessions 
in evaluation information and procedures. ^ ' ^ 

TABLE IV- 3: "TRADITIONAL" .INSERVICE SESSIONS CONDUCTED BY O.R.E. STAFF 

DURING 1975-76 

LENGTH OF 

TARGET GROUP ^ - TOPICS ' SESSION 



All Title I Principals 



All Elementary Area 
Directors and Coordi- 
nators n 

All Title VII Bilingual 
Program, Principals,- 
and Teaching Staffs' 



School-level data availabler^or use in j 1 week 
setting school gpals; The CIPO planning • 

V and evaluation model; How to plan a 
s chad- specific Title I program 

The CIPO planning and evaluation ;nodel; 1 day 

How to interpret standardized test ^ 
scores 

What is Research?; How to* plan a research- 2-3 hours 
based Title VII program 



^ J 

One alternative identifiey was 
sent to all ^ISD instructional 
tion (Feedback ) was to share 
of O'.R.E. Each month the news 
the evaluation of' the new quar 
etc. However,^ there were regu 
a listing of all O.R.E* report 
Frada" - a^ question and answer 



the creation of a montl}ly newsletter. which was , 
and clerical staff. The purpose of' this publica- 
information about evaluation and research activities 
letter featured one of O.R,E. 's activities, e.g., 
ter system, the expanded achievement testing program, 
lar "columns", 'including the monthly testing schedule, 
s and' documents to .be completed that mont^ and "Dear 
column written by the O.R.E. coordinator. ^ 



At the end of ^1975-76 school year. Feedback had a circulation of 5,500 persons: 
In addition to AISD staff, it was mailed^ to all University of Texas education 
faculty, the major city public school R and E directors, the local media and 
selected national, state, and local officials. The printing costs for 8 issues 
are estimated to have been around $1,100. . It required around 320 hours <40 days) 
of writing and editing time and about 96 hours" (12; days) of= secretarial time to 
mall the newsletters. The personnel costs were indeed high,^^but considered , 
well worth it for this year. .The newsletter was received well by AISD s-taff ^ ' 
i.e., only positive written and vferbal compnents were re^ceiyed by the editors. 
A survey bf 300 AISD' staf|, members is planned fos late August, 1976, to ascertain 
whether or not Feedback 's, continuing existence is warranted. ^ 



s. 
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Another new approach which has been successful in some situations this year was x 
the use'of brochures and pamphlets- to communitate O.R.E, information ^nd proce- V 
dures. Three such brochures were«developed this year: 

The New Quar^:er Systemr What Parents, Teache|*s, and Studeiks Think 
Title VII Bilingual Research in AISD . , , .> . ' . 

^ The Office of Research and Evaluation: Providing Information for Decision 
Makers ... ' . 

This approach, like sli^e-tape shows, work^some of the time with some people, 
^They are successful as accompaniments to papers and presentations, as follow-ups 
to nipetings, and with' persons who.already want the information and/or do not 
havel time to read detailed repor/s.' 

A^Vord of caution: brochures, pamphlets, and newsletters could easily become v * 
just as invisible as memos and' reports if used with great regularity. Like modt 
effective mass communication, the attention-getting potential of a coitimunicati'bn 
is* determined by its novelty or newness. ^ 

Another alternative to, inservire, especially at the* top level of decision-making, 
was the involvement of 'key people in making presentations on O.R.E, philosophy, - 
procedures, or findings. Quite frankly, thi^altemative was not premeditated.' 
It is' recorded here merely to document what turned out to be a successful method 
of communication with very, busy, and time-pressured decision-makers. 

Another alternative which was identified but which never came to fruition for . 
lack, of time resources ^ was the writing of a ^chool Based Planning Matiual . 
This manual was conceptual'ized as a planniifflg handbook for principals* use iit 
setting school goals, for progtam staff to/use in' planning federal programs, etc. 
Their involvemeng^^n its development phas/ would be crucial to eventual use; time 
pressures on these key people as well as X).R.E. staff made postpbnement appear 
advisable. However, awriting task force of AISD staff experienced in planning 
and who were also familiar with the ClPO planning and evaluation' model was organ- 
ized.- This task force did much initial work, all of which is on file in O.R.E. 
should the needed time Resources become available in the future to complete this 
manual. ^ . 

Another alternative which was used ,with great success was the "training package 
approach." This approach wa3 used in situations where the target populations 
were too largexand/or scattered to be ^trained ^directly by O.R.E. staff. In 
these cases, particularly in the area, of testing procedures and testing results, 
0»R.E. developed "canned" inservice packets including a detailed script^ trans- 
parencies, handouts, practice materials, etc. These.materials were then modeled 
by^.Jl.E. st^^f, and in one case the Superintendent, -i'or audiences of trainers 
(principals, instructional coordinato^rs, teachers,^ etc.) .> These/traiijers then 
received complete copies of the training packet and used it later with more'local- 
ized groups, usually at the building or community level. The most comnfete sets- 
of training packages dev:eloped by^ , Title III staff this year are: 

Sharing School Level Test Scores with Your Community 

Getting Your Students Ready for Standardized Testing (Elementary Level) 

The laat alternative to traditional inservice used by O.R.E^ this .year to promote 
understandfag of evaluation was. not new. It simply involved "being a consal£ant" 
to ^hosj? individuals and schools who requestejd evaluation services from the office. 



This meant that when a School asked for, assistance in setting school goals, the 
responding.'o.R.E^ staff person. taught that school faculty the componei^its of the 
CIPO plannitfg and* evaluation model,' leading them to set their own goals and to 
develop their own evaluation strategies. When a federal project staff asked for 
assistance in designing project objectives ^hd evaluation designs for the coming 
year, the O.R^E. consultant worked to increase understanding of the relationship 
between project outcomes , processes , and inputs *. 

The above seven techniques all demonstrated or promised some success in our staff 
development efforts this year. But we wfll have to ^ind many ^ other approaches 
to achieve our overall goal of "promoting understanding of evaluation information'^ 
and procedure among AISD stalff.'! Proselytizing for rational planning-^^nd res- 
ponsible evaluation can be a lonely and frustrating expegrience in an Environment 
where unexpected court orders, legal holds, and late f-^ding are regular occur- 
rences. Just as students "in public schools do not all learn the .same^way fVom 
the same approach, neither can CQiranunication with teachers, principals, adTn;t.n- 
istrators, and board members be standardized, the search f or effective means 
»of communication will continye in our woj-k for years to come, 

0«R.E. will draft and revise an inservj,ce plan for 19)5-76 subject to continuous 
review and updating . * ^ ^ 

The Title III evaluation project's inservice plan was drawn up in late September, 
1975, and it was 'continually reviewed and updated. A copy is available on file 
in O.R.E. for review. ' - 

O.R,E. will acquire and create inservice materials in accord with the inservice 
plan developed J ^ * 

' X ^ \. 

We^ anticipated that few evaluation materials were available on tj^g^market ,, but 
we were supriz^d to learn how few existed. Consequently, all of the inservice 
materials used by O.R.E. this year were "home grown" products. This included 
the training packages referred to in the previous section, and exercises developed 
for the CIPO model orientation inservices. 

O.R.E. will provideMnservice sessions in accord with the inservice ptan developed . 

Specific instances of the inservices and alternative^ used by O.R.E. this year . 
have already been documented in the 'first subsection of the Objective 3 activities* 

„• / 

(OBJECTIVE) O.R.E. will institutionalize the AISD\CIP0 evaluation model. 
LEVE;L of ATTAINMENT: Not Achieved. 

4 t 

This is^ andther one of those areas in which absolute attainment is not jtrobable 
for several' years, if ever, and although all activities ^proposed to achieve tb±i 
objective were completed , 'tt must ^e conceded that only Initial ^ stfeps have been v 
taken in this area. * • ' 




SUPPORTIVE DATA: - ' ^ ' ■ ' ^ ^ i/ 

O.R.E. will Implement proposed slgnoff procedures (approvals) assuring that eval- 
uation plans in accord with i^t^ CIPO model are integral to all prQposed major 
"district projects . 

This year,- the proj'ect director developed and submitted to the Superintendent 
^ for his review several O.R.E. policies and procedures. Among several of these 
documents "review (of) evaluation designs by appropriate personnel" ( Participation 
in School/Program Evaluation ) is indicated. However, as indicated on the' first page 
6f this chapter, there are problems associated with required signoffs. The future 
of "signoffs" on evaluation is at this point really jjfet certain. ^ 

O.R.E. will assist central adipiinis'trarors in the selection and assignment of eval- 
uation projects for the AlSD/evaluation unit for formal evaluation in the 1976-7? 
school year . ^ ^v, - / ' \ ^ " 

The selection of 1976-77 ^dl^trict evaluation priorities was performed in three 
phases. The first phajsTe involved collecting through group meetings and inter- 
views evaluation 'priQtities from a) Central Office Directors, b) Board Members, 
and c) the EValuat^n* Advisory' Committee. The areas suggested by these groups 
were then compll^ inj^^ one list with an indication of the frequency with which 
they had beea »^ggeslt§9 , 

<Ln the secorW phase this list was^bmitte'd to the Superintendent who with the 
help of the Cabinet selected -several areas for final consideration by the Board. 

The th/rd, phase was carried out in a Board meeting held ^om 10:00 a.m. - 3:jp0 p.m* 
on Saturday, May 1, 1976. The Cabinet, the meViia, and the O.R.E. Coordinator also 
.att^ded the meeting whichliad the twofold purpose of selecting district goals 
»anjl selecting district evaluation priorities for 1976-^77.- A complete packet of 
introductory materials, worksheets, and exercises used in that meeting by the 
O.R.E» Coordinator is on file in O.R.E. for interested readers. ' * 

The final decisions made by the Board at the May 1 meeting are shown below: 

' 1976-77 District Emphases 

Achlaveuient in Basic Skills / 
Achievement of Low Socioeconomic St'udeiits 

Testing of Teacher Basiti Skills <Math, Reading, Grarmay{ etc.) 
Desegregation 

/ 

1976-77 Evaluation Priorities 



Achievement of Low Socioeconomic Students 

The Quarter System (Second^^ Curriculum Evalua/ion) - Yeat Two 



0 




(OBJECTIVE) O.R.'E'. wil» adTninl,ster *and/or coordinate evaluation activities of 
the district. " , ^ - . 

LEVEL OF ATTAINMENT: Achieved. . . ^ ' ^ \ ' ' \ ^ " . ' 

The activities proposed tp attain this objective have been carried out. \ ' 
SUPPORTIVE DATA: ^ . ^ . ^ . ' \A 

VR>E. w ill prepare evaluation designs for projects selected for^evaluation ' 
and\ will review these designs with/approi5rigte decision-makers . 

^ .O.R.'E. will execute these, design plans . . . 

O.R.E. will prepare and present f ormatlve^ evaluat^ic|l' reports ^ 

O.R.E> will prepare and present summ^ive evaluatidti reports . 

Each of the above 'four activities are reported on under Objective 1 in the first 
several pages of thia chapter. Each of the ^out' activities w^ire carried (»ut. 

O.R.E; will pr ovide consultation, ^te.chnical assistance and/or administration on 
evaluation designs and evaluatioTIg"iEor federal programs . ] ^ ' 

Early during the 1975-76 ^t<hool year, O.R.E. employed a senior level evaluator 
through district monies to provide evaluation services to district personnel 
on a request basis. One of the requests to our offic% for ^assistance this 
year came from ESA4 staff who were preparing 'the Basic ^nd Pi-lot proposals 
for- 1976-77. It was estimated that 120 senior evaluabor hours and 80 intern- hours 
were expended in assisting the ESAA staff to forraulatfe program objectives an4 . 
in writing evaluation strategies for -th^ proposed project. 

The Title VII and the Tittle B^^Tftoosals for 1976-77 were also the ijroduct of'' 
•much O.R.E.^ staff input, speci^f icklly in terras of 'clarification of ^ol>jectives with 
p;:ogram staff and in proposing evaluation strategies t?o evaluate the achieve^ 
ment of those -objectives. . " * ^ ' ^ 

The Coordinator of O.R*E. spent several days^ time offering advice on-prograip 
design and e^JaIuation strategies .to AISD staff who were preparing Title IV 
proposals for 1976-77 funding. . . . . i 

^ . ' ' ■ * 

O.R.E> will monitor, all O.R. E. -contracted evaluation efforts . ' ' ^ , 

t 

There were pnly three O.R.E. -contract^ evaluations by outside consultants *in 1975-76, 
The largest one was granted to ARBEC/(Appiied Research for Business, Education, ^ 
and Community Sey^ie'e) for the evalt|atio'n of Austin ^s ESAA Basic grant. . .-"^ 

i).R.E. assigned one of its s.eniot ¥v^aluation staff members to oyersee this » 
specific contract with ARBEC.,, The tasks involved '^rior to the award of this 
contract were discovered to be: " . , 



PHASE ONE - Prior' 'to C(^trag£ award ; " . ' 

1. Develop an RFP (Request for Proposals).^ 

2. Develop a list of bidders, . " ' ' . • ^ / * • • , ^ 
3- / Send out; RFP's.' * \ , > * '^^ * \ \ 
4/ Hold bidder's meeting to clarify RFP. - " - ^ 
5, Establish district readers committee to review incoming bids, 
6; Design rating form to us^ in rating submitted bd>d§»' 
7; Oversfee reading and rating\)f bids submittedT 

8. Summarize ratings. ^ ^ . \^ * 

9^ Hold decision-making 'session with reading/ committee to come ?to' concensus* 
" concerning award of bl^. • , * ^ 

10. Notify winner (and losers) of bid, . ' ' 

11, Award contract. * - ^ . ' ' . l.<^ 

During this first phase, it was estimated that t^ senior evaluator iiv^krge 
spent 11/2 days per week for a two month period, wording with both program 
personnel, district rea-^tjers, ' and bidders. Phase One wa^ seen by O-R.E. as a 
surprijirigly expensive strain on O.R.E. time resources. 

PhasgTwa of this activity, occurring a£ter the -a ward of the cori tract involved: 

f PHASE TWd ^ Subsequei:^ to contract award and during - - 

the perj-od of the contract ; ' * J . 

' t . ^ / . , . ' , 

1. Visit vendbr's site of opeVations to monitor work ^ clone; land in ^progress. 
^ Communicate ^nfoVma^ion betw^een vendor and program personnel, ' 

Arbitrate disagreements over evaluation* procedures ^ etc. , between vendor , t 

'and program personnel^ - \' ^ . . ' ' ' 

Perform other liason/communicatioA activities between vendor and program 
personnel as were required JuriiJg' the^ perio'd pf the contract. 

*^ , . / ' , 

The senior evaluator estima:ted that* 1/.2 day every other, week of his tinje fo*r^ " 
the duration of the contract (16 weeks) wae required^ to conduct the activiti^ 
of Phase Two. . 

'Considering only the time Invested by one senior evaluator. and one O.R.E. 
secretary, the O.R.E. personnel costs for overseeing this evaluation project. were 

21 days Senior Evaluator @ $73.00/day . ' ,$1,533.00 

* 5 days Seci:etary (? $25.0'o/day ' ^ 125. DO ^ 

/Total • . ♦ ' $1,658.00 ■ 

^"^^ese cifst'^were supplied by AISD district funds,, and^were not supplemented by 
•ally ESAA monies. . ... 

" ^ ' ■ I". 'X"- 

The other two contracte^d evaluations for the ESAA Pilot projact were monitored 
by a halftime evaluation intern who was also responsible for^conduGting an out- 
come evaluation of that project, therefore, the majority of the costs of over- 

^- seeing these two contracts were covered 'by pSAA' *fuhds» '/ ' ' ■ 
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O.R.E, will pro vide consultation and technical services or assistance on 
evaluation to MSP staff persons . ^ ; 1 

^Thro^^ghout the y^ar, many requests for evaluation servlx:es were made of O.R.E. 
.personnel, e.g.^make presentations to. school faculties, or P.T.A. groups, 
provide Inservicfe to school faculty *prlor. to school^level goal setting by ' 
principal and teachers, * etc. At the beginning (>f the year, a propedure was 
established within the office for recording the number and extent of requested/ 
services to O.R.E. This procedure Involved simply filling out and processing a 
Service Request Form". As the year progressec^ O.R.E. personnel becamefeegli- 
gent in filling out these forms as servjfces were requested of them and simply 
supplied' the service to the requesting Individual or school^" However , one * 
senior evaluator did keep a rather complete record pf seWices requested of 
him. These requests are summarized ^in Table IV-4. When queried regarding the 
accuracy of the time estimates entered on these service request forms, the eval- . 
uator indiqated that the time alld costs of meeting these requests probably were 
underestimated by about about half. AH .evaluators who were federally funded (and 
thus prevented by guidelines from responding to requests from persons not employed 
by some project) performed substantial numbers of comparable service activities 
for their re'Spective program staffs. . " 



TABLE IV-4: SAMPLE OF SERVICES REQUESTED OF O.R.E. BY aISD 



NAME OF REQUESTING DEPT. 
PROGRAM OR PROJECT 

1. Developmental Program 
Title I Migrant 



O.R.E. RESOURCES REQUIRED: 
Evaluator - 3 hrs. 
•Clerical - 3 hrs. ' 



TOTAL COST: $37.65. 



DESCRIPTION OF SERVICE 



Mean CAT test scdres for 
identified migrant stu- 
deiit? in Gradies* 1-6 in \, 
identified Title I Migrant 
Schools (scqres to^ be for 
1976); this "'^.'s to be used 
for a Needs Assessment to ^ 
prepare an application for 
the Title I Migrant Sxxm^ 
School Program. 



STAFF DURING IS 75-76 

_END PRODUCT DESI RED 
\ 

A listing, by grade, 
of the Mean 1976 
California Achieve- 
ment Tes-t scores for 
^identified Title. I 
migrant students in 
Grades 1-6,' reported 
in Grade Equivalents. 



ReiUy EV S. 
Pilot Project: 
Project Increase 



O^^.E. RESOURCES REQUIRED: 
Evaluator - 4 hrs. 



Meet with principal to 
help plan projecfe.^(1.5 
hours) apd write a dr^ft 
of needs, oSject Ives, ac- 
tivities, evaluation sec- 
tion (2.4 hours). 



Draft of prqposal 
sections. 



TOTAL COST: $37.00 




TABLE IV-4 (cont.) 

OF REQUESTING DEPT.^ 
OR PROJECT 



DESCRIPTION OF SERVIC 
REQUIRED 



3. /Area I Asst. Director 
Area I Staff Dev. 



O.R.E. RESOURCES REQUIRED: 
Evaluator - 4 hrs. 
Clerical - 2 1irs.. 



TOTAL COST: $46.00 



4. Williams E, S. 

Williams E. S. Parent 
Survey 



O^R.E. RESOURCES REQUIRED: 
Evaluator - 2^ hrs. 



TOTAL COST: $220.00 



5. Planning and Program- 
ming Data Services 



i 



O.R-E. RESOURCES REQUIRED? 
Evaluator -'l60 hr&. 
Data Specialist - 8 hrs 
Clerical - 24 hrsi 



TOTAL t:OST : ^$1, 200. 00 




END PRODUCT DESIRED 



Pr^are and present a talk 
on Esc Xtll Staff Develop- 
ment Need^ Assessment 
Survey • ^ 




Talk given. , 



Meet with principal. to 
plan the conduct -of the 
survey; to assist in 
writing items and for- 
mating instrument; and 
to ^ssist ia\interpret- 
ing the data. 




NA (PHncipal, UT 
prof essolss, and ESC 
XIII resport^ble for 
tangible pro^\i^ts) . 



Select /develop"^ a measure ' 
of (a.) "User ^Percept ion- 
of Service Quality" and 
(b.) '''Staff Perception of 
Effective Organizational^ 
Functioning." Collect 
user and staff baseline 
data and write report for 
manager. ' 



Written report in 
partial .fulfillment 
of boa^d*s request 
for ifianager to 
report back^.on 
changes in Data Ser-r 
vices . Repo;rt'^is 
seen as a brief doc- 
ument that will allow 
manager and the dis- 
trict to decide if 
any further alterna- 
tive' actions, need to 
be 'considered at 
this time. 
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TABLE IV-4 (cont.) 

NAME OF REQUESTING DEPT. 
'PROGRAM OR PROJECT , 



Department of School 
and Community Rela-, 
tions • CookN Commun- 
ity Schoo»l (Community 
Ed. Office) 



O.R.E. RESOURCE^ REQUIRED: 
Evaluato^ - 2 hrs. 
Clerical - 2 hrs. 



DESCRIPTION OF SERVICE 
REQUIRED 



END PRODUCT DESIRED 



Discussion of Cook Commun- 
ity School Objectives of 
the Advisory Board* • 2 hrs. 
Writing- Objectives - 3 hrs. 
Presentation to Board and 
follow-up - 3 hrs. ^ 



Written Ojbjectives 
for Community School. 



TOmL COST: $50.00 



7. Travis H. S. 
Project PAVE 



O.R.E. RESOURCES REQUIRED: 
Evaluator - 48 hrs. 
Clerical - 12 hrs. 



TOTAL COST: $500.00 



Consultation and direct . 
service ^n developing an 
evaluation design compa- 
tible with the AISD CIPO 
model (1 man-day each^Nov- 
ember, December, January) . - 
arid assistance (to ^average 
.5 man-days February-July 
,in selecting consultants, 
designing instruments for 
site visits I and monitoring 
production of the evaluation, 
report. * 



(1) Evaluation de- 
sign acceptable to 
Project PAVE., 

(2) ' Eyaluation ^report *• 
Note: The actual con- 
duct of the evaluation 
will be performed by.^ 
ESC XIII which will ^ 
{provide at Iteast 14 
man-days service and 

be partially reimbuf^ed 
at $500 fronr Project 
PAVE consultation: 



r 



(OBJECTIVE) O.R.E. will disseminate Titie^'tlT^project information3r state and 
national levels. 



LEVEL OK^ATTAINMENT: Achieved. 

All four mijdr activity areas proposed in the Title III proposal ,for meeting 
this^obj'ectfive n^ve been fulfilled. 

SUPPORTIVE DATA: " * 

O.R.E. will solicit, prepare, and/or present at least one project report at a 
national level meeting^ \^ ^ ~ / 

This activity was',, if ^^ny thing, %verachieved. Not one, but four presentations 
about Title III prbject inf ormatiott were made at national meetings. 



The first presentation was a paper presented by AISD Superintende.nt Dr. Jack Davidson 
at the February, 1976; annual meeting of the American Association of School Admin-, 
Isttators (AASA) in Atlantic City, New Jersey. The title of this papet was '-'The . 
Research and Evaluation Unit: Helping Your School Board Make Desisions." * 




A second presentation was made >y Dr. Freda Holley and Dr^. Ann Lee at the annual, 
meeting of the American Educational Research Association (AERA) in San Francisco', 
California, in April, 1§?S\ This paper was entitled "Models for the Delivery of 
School District Evaluation: Service or Accountability?,. ^' 

A third presentation wa^ also made at AERA la San Francisco in April by. Dr^ Ann Lee, 
^.R.E. /Title III staff member) in the form of an audio-visiial presentation entitled 
"Research and Evaluation in lublic Schools." <ffeis slide show^resented a generic 
view of p^ttblic school R and X units and'whaH they db. It described the activities 
ofJi^Le Austin ISD^s Office of Researcl^ and Evaluation, depicy.ng many facets _gf " 
the Qf,f ice's operation which have bfeen seeded by Title III monies granted to the » 
district over the last three years. - - * 

Tri^fourth presentation, also at AERA *was fiSSeSy^. Paula Matuszek. This*paper» 
.described the results of an evaluation of the 1974-75 Austin ISD Pupil/Teacher 
Ratio Reduction Program. Dr. Matus^f^k had conducted this study while empldyed 
by Title III funds in p.R."E. during the 1974-75 school year. ' 

O.R.E. will solic it, prepare, and present at least one pr1)iect report at ^he 
state levels I \ ' ' ^ 



/ 




This^;pbjecEive was met by an O.R.E. -coordinated pane^ presen^tfCT^rST the Texas 
Association of School B^ards/TexasJ^sociatl^^h ^of School Adnttnistrators annual ^ . 
meeting in San Antonio, -Texas, in^tember^ 1975. The presentation was entitlecU' 
Research and Evaluation Units: Helping School Board£JJake Decisions". . ' ' ^ 

The panel consisted of: , . • ^ 

Will B. Davis, School Board Member, Austin, Texas, 'and President of TASB ' 
Dr. Freda Holley, Coordinator of the.Office of Research and Evaluation, 
Austin I.S.D., Austin, Texas S . . f . 

Nancy 'iuay, Si:hool Board- Member , Dallas I.lS.D.", Dallas Texas 
Dr.- William Webster, Associate Superintendent, Research, JEvaluation, and , 

' P4anning, Dallas I.S.D., Dallas, Texas; ^ ^ . 

Moderator: Dr. Ann Lee, Office of Resear* and Evaluation, Austii* I/S.D. , 
Austin, Texas ^ ' . • o 

The panel was presented in two- time slbts. A tdtal of about 400 persoh-s att6nded ' ' 
efther one or the othe^^ of the two sessions. The fitst pa?f^ of each session con- ' ' 
sisted of ;:he panel answering prearranged questions likft "What kind of information - 
can research and evaluation units, collect?" "Whef| can a district get the resources - 
to start a research and evaluation. unit?" "What fil^isijpnfe do school bdards.have 
to make for which research and.^v^luation units can 'provide Information?" The.' second 
phase- of each session was./a question and answer period during which members oT the 
audience addressed questions to panelists. ' ■ ^ ^ * 




The Joint Urban Evaluation Council will meet five times during the year to 
share evaluation information and to engage in cooperative ventures . 

The Council actually met six times during 1975^76 to share evaluation information 
.(see Table^-IV-5 below), - ' \ ) 

TABLE IV-5: TEXAS JOINT URBAN EVALUATION, COUNCIL 1975-76 ACTIVITIES 





MEETING DATES 


MEEi^ING PLACES 


TOPICS COVERED - 


1. Sept.- 25 & 
26, 1975 


Austin, Texas 


Reports from each of the 7, districts 
on evaluations conducted in tTieir 
districts. 

TEA and accountability.' 

Plans for coming year. " " - 


2. Dec. 4 & 5," 
1975 

* 

<* • 


El Paso, Texas 


JUEC position paper on accountability- 
past and. future position papers. 
Accountability Legislation — past and 
present. 

Up^coming evaluation topics in the 7 
districts.^ * ' 
-Review of Title IV proposal. Project 
META. ' -'^ 


' 3. ^an. 29 & 
30, 1976 

♦ 


Houston, Texas 


Reports 'from each of the 7 districts ^ 
on^evaliiation in progress in each dis- 
trict. 

JUEC Title IV project discussion. 
Miscellaneouj^-* 


4. March ,18 & 
^ 19, 1976 • 


San Antonio, Texas 


Reports fronr^'each of the 7 d^istricts 
on evaluation -reportip released re- 
cently in their districts. - ,i 
The new accreditation scheme by TEA. 1b 
Next' yearns Title IV JUEC Project || 
. META. ^ 


5. 4pril 12-13', 
1976 


Houston,' Texas 
• (Expenses bom 
by ^IM^trljts) 


Met with Chief State Instructional 
Officers to discuss potential account- 
bility legislation and to define a 
common urban approach to the, new state 
accreditation' model. 


6. May 12, 13, 
- 14, 1976 

' ' . . * 


Fort Worth, Texas 
• 


Review of JUEC statement to -legisla- 
tive subcommittees on accountability 
legislation. . . 
Election, of new JUEC officers. 
Minimum Competency Testing - yes or 
no? 

One/half day joint meeting with thief 
State Instructional Officers — account- 
ability in ttie seven urban districts. 
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' This thiird year of the Council's existence has been marked by strengthened commu- 
ndhations among the' members ^between regular meetings. There has also been a grow- 
filSgr awareness on the part of the s^ven cities' superintendents and curriculum 
^.^c^V'i. ^hlefs that the Council members have much information to provide in statewide 

li^cision-making situations involving these groups. For example, the Council met 
twice in joint sessions with the Chief Instructional Offfcers to disctfss cooper- 
ation among the seven urban districts on upcoming state accreditation procedures 
and on possible competency level testing procedures. In fact', the latter of these 
j.oint meetings resulted in a Title IV, Part C application to the Texas Education 
'Agency for funds to design a competency l^vel test appropriate for students in 
the geven large Texas cities. (At the publication of this report, it is.not 
known whether or not this Title IV application will be approved by TEA.) 

Much of the credit for the success of the Council and of past and future contri- 
butions it has made in providing' informatiorf for -decision-making on an intercity 
basis is due to the hard work of several staff members of the Texas.'*Baucation 
Agency, particularly Dr. Walter Howard, Dr. Dorothy Davidson," Dr. HaJ||ii Ford, 
Mr. James Hill, and Dr. Maurice Dutton (no ionger with the Agency). *THese indivi- 
duals, especially Di^. Howard, have spent hundreds of hours over the past three 
years interacting with the Council. members. Of inestimable value has been the 
information shared by these and othef TEA officials with R & E 'staff members at ' 
Council meetings. 

Earlier in the year^December, 1975) the Council wrote a proposal and was sub- 
sequently funded for a Title IV, Part C project from the Texas Education Agency. 
This project will operate from July 1, 1976, 'to June 30, 1977 (or 1978). Project 
META,- Materials for Educational gaining in Accountability is summarized below: 
S ''^ . , 

This project addresses the state priority of accountability . Ther^ 
is a strong need V or educational evaluation research information to be 
«more effectively used by educational decision-imakeirs to improve public 
school programs. The Texas .Joint Urban, Evaluation Council will address 
this need through planning, developing, piloting, and refining training ' 
materials designed to train the users of e'ducational research information' 
(teachers, administrators, and board members) how to more effectively use ' . 
evaluation research information for the improvement of schools. 1 

The major objectives of Project flETA during 1976-77 are: 

(1) The training needs of Texas educators 'in the area of more 
effective utilization o|^evaiuation research will be identifi.e.d, 

' ^ an^ the training topics and formats will be planned. *^ * ' 

(2) Training approaches will ^'developed to help evaluators imp^rove 
the utilizatioiv<^f evaluation research by decision-makers. 



Ijoint Urban Eyaluatibn Council'a Title IVi Part^ C Application to the Texas 
Education Agepcy (Project Abstract) for^ Project META, Materials for Educational 
Training in Accountability. December, 1976. * 
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(3) Guidelines and training '^^41 be developed for evaluators, to ^ 

help them make evaluation "reseaMh information more understandable 
• and communicative to decisioti-maKers. ' . 

C4) Project training materials doyeloped will be disseminated on a ' 
-local, state, and* national, level, as measured b/ project records,-"- 

Austin's Office of Research and Evaluation will serve as the fiscal agent of 

the Council Muring the ope:;3tIon of Project MEXA. The grant was awarded in 

the sum of $67,588. ' " - • 

O.R.E. will write and disseminate on a statewide basisjtwo papers on; (1) models 
for the -delivery of services to school districts, and (2) on potential problems 
encounte red in the establishment of school district evaluation services. Draf"ts 
of these papers wil l be reviewed with Texas Education Agency prior to publication . 

These activities were performed in a variety of ways. The first paper, "Models 
for the Delivery of School- District Evaluation: 'Service or Acdtountabili'ty'" 
was delivered -by the Ti^le III Project Director," Dr. Freda Holley, and Title 
III ^enior Evaluator, Dr. Ann Lee, at the annual meeting, of the American Ed- 
ucational Rea^^ch "Association in San Francisco, California, in April, 1976. 

'A subcommitt^ee . of the Joint Urban Evaluation Council prepared a position papeT 
for the Council, "A Position Paper and Recommendations Proposed by the Tfexas Joint 
Urban Evaluation Council For a State Supported Research and Evaluation System For 
Texas School. Districts This posttiorf paper along with .statements of testimony 
were presented by invitation to the following committees and subcommittees of the 
T^as Legislature during the 1975-76 school year: 

Subcommittee/of the Texas House Public Education Committee ofi^Goals 
for Public Education - April 20, 1976 ^ ' * 

^ Joint T,exas Legislative Committee on Intergovernmental Operations, 
Renewal and Accountability Systems - May 14, 1976 ^ 

A paper on potential problems encountered in the .establishment* of school district^ 
evaluation services was prepared by Dr. Frfeda Holley, JUEC chairman and Title III 
project director, and^was presented to a subcommittee of the Texas House Public. 
Education Committee on Goals for Public Education on April 20, 1976, by Austin^s 
School Bo^rd President, Carol© Keeton McClellan, ■ ^ 



%oln.t*Urbab Evaluation Com^^iTts Title IV, Part C Application to the Texas 
Education Agency (Project iCbstract) for Project META, Materials for Educational 
graining in Accountability. -^Deceniber, ■1976. " . • 7 ► 
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(OBJECTIVE) .O.R.E. will perform efficiently all activities designed to achieve 
the other project objectives outlined. t ' 

V . . ■■ * ^ • ' 

• LEVEL OF ATTAINMENT: Achieved.'^ • . . • 

* ^ • 

The twenty- five activitsies proposed to achieve the previous six Titie III 
•evaluation project objottives were all carried .out • 

SUPPORTIVE DATA:. - - 

The Wiect direct or will provide project management and fulfillment of reporting 
requirements for the Tjtle III, project . ' ^ 

The project director' (as coordinator ..of O.R.E.) was able to oversee the projecf's" 
activities. All planned evaluations , were carried out to- completion 'by the end 
of the xear. She wrote the evaluatioh policies and procedures and submitted^them 
-to the Superintendent for his re'View", " supervised the testing program, helped » 
'to plan and supervised the evaluation dissemination and inservice subcomponent ^' 
met with the School Board to assist them in setting 1976-77 district priorities, 
provided technical assistance on 1976-77 .evaluation designs" for both local and 
Federal programs, and secured state, and national audiences .for O.R.E. presenta- 
tions op. the Title III evaluation project. Regarding r-eporting ceauirements of 
the Title III project, she contributed to and reviewed this reportV-^ 

0..,R.E. will provide inservice training for its staff .(includ ing professional 
meetings) . . \ 10^ \ ■ . ". 

Five of the seven senior evaluation staff attei^ed "the annual meeting of the 
American -Educational Research Association in San Francisco in April, 1976 
tonly two were supported on th±s^:trip by ;ritle III funda^. This meeting is viewed 
as the- most important professional meeting of the year for O.R.E. senior s1;aff 
members. It' iPperhaps of interest that' O.R.E. coordinator Dr. Holley is program 
chairman of AERA's D^ision H (School Eyaluation and Program Development) for 
the 1977 annual meetHg. _ The award of this responsibility is evidence' of the 
national reputat^n which O.R^E. has earned in less than thre^ y^ars of operation. 

O.R.E. staff attehldance at other professional meetings was also enpouraged when' 
it was appropriate. The following table summarizes staff attendance .and parti- - 
cipation at various conferences during 1975-76: 



TABLE .IV-6 : 



O.R.E. STAFF PARTICIPATION IN PROFESSIONAL GROWTH ACTIVITIES 
' TOJRtNG 1975-76 * 



CONFERENCE MEETING 



N0» OF.O»R.E» ' 

STAFF ATTENDING 



m. OF PAPERS OR 
PRESENTATIONS GIVEN 



1. , National St:aff Development Cdtiference 
Council (Austin, Texas) 

2^ Texas Association of School Boards/ 

Texas Associatiohi pf School^ Admdi- ' < 
, istratprs Annual Meeting (San An- 
tonio, Texas) 



TABLE IV-6 (cont4 



CONFERENCE MEETING 



3. 



American Association for Supervision 
and Curriculum Development (Miami, 
Florida) 



4. International Bilingual/Biculturai ' 

Education Conference' (San Antonio, 
Texas) ^ ^ - 

5. Texas School Public Relations Asso*- 

cia^ion Winter Seminar (Austin, 
Texas) , ' ' • . , 

6. Statewide Conference on Evaluation 

of Title VIl' Projects (Waco, Texas> 

7. Education Service Ceriter XIII Confer- 

ence oa Achievement Testing (Austin, 
.Texas) - . -J 

8^ American Educational Research Assoc- 
. ' iation Annual Meetipg- (San Francis- 
co, California) ^ ^ 

9. Five meetings of the Texas Joint 
Urban Evaluation Council 

. Total Number of Papers or Presenta- 
tions Given by O^R.E. during*1975- 
76 at State or* National Profession- 
al Meetings 



NO/ OF O.R.E. . NO; 0? PAPERS 
STAFF ATTENDING PRESENTATIONS 




OR 

GIVEN 



2 



if ,3 



N.A.* 



.•11 



*^^^..T^7. O.R E st^ff member ^attended at least one of the 1975-76 JUEc' 
W ^^f °^ these .meetings. O.f.E. reported on work-in progress, 

but they wer,e not papers as such. * h uBicat,, 



GLOSSARY 



1." Affective behavior - Those learning .vajl^es' dealin^rimarily with motivation, 

feelings, attitu/es, interests, appreciations and 
' values. " . . ^ • 

, 2.^ 'Assessment - The process by which data, are ^gathered and used for systema^c 
descriptive and .predictive purposes with respect to a person, 
^ group, organization, and/c^r some other object being studied or 
^ ^ examined. / The results of assessment often provide the^data ancl 

V V. information needed for ^valuation. 

•3. Boehm Test of Basic Concepts - Fifty items arranged in order of their 

difficulty. Each item consists <^a set of 
pictures about Vhich st^ements are read to 
the students •(usually in kindergarten)^ These 
statements briefly describe, the pictures and ^ 
ask the child to mark the one illustrating 
the concept area. ... - 

4. California Achievement Tests (CAT) -A standardized achievement test battery 

' ' ' with noirms. , It covers Mathematics and 

' . * Verba VCompxehens ion areas^.- The test^ 

^ / ' provides assessment; of achievement levels 

^ - • I comparable to percentiles on a national 

^ ' level. , 

5. CIPO - An evaluatiotf model designed by the AISD Office of Researcfi arid 

Evaluation. Its purpose is to provide procedures for evaluating 
the special programs of the. district through use of inputs, processes,' 
and outcomes concepts. Th^ evaluation is aimed at providi^ig informa- 
tion to decision makers. > . ^ 

f 

f 6, Decision making - .Making choices ^mong alternatives. 

\ 1. Decision Questions - A formulation of decision. alternatives or explorations 

for (Jecisidn alternatives in interrogative terms which 
can serve as formats for the later presentation of - 
decision recommendations based on evaluation information. 

8., Evaluation, Context - Evaluating the planning of the program in gelation to 

the^ environment in whidi it. will or did operate. 

9>f^^aluation Design - A predetermined strategy for judging the effects of a 
; \ \ . \ ' program. ' ' , * 

10. Evaluatioix*^ inp^t^ - ^alysfs^jE thV resource's available to a program. 

11» 'E^aXu'ation, Process - Judging the if fectivenesB of the activities and pro-' 
*• , cedures of a program. * " / 



12. Evaluation/ Outcome -"^ Judging the effectiveness of the products of a program 

*(in.the CIPO model, this means student behaviors. >\^' 
« . > 

13. Jo itv5' Urban Evaluation Council (JUfiCl - A coramit.tee" consisting of representa- 

tives from* the 7 largest urban school 
^ * ddst^ricts in Texas. The purpose is to 

^ • share evaluation information and ideas ,j 

and to consider "^evaluation problems 
• and goals of evaluation in these-. 

ricts . 

14. ^ Median - The middle score or number in a distribution. 

15. Outcomes - The term refers ..to any student behavior whether affective, 

cognitive, or psychomotor, whether covert or overt, intended or 
unintended. 

16. Percentile - A score below which. k given- percentage of scores lie." 

17. Priority - What activity or item to which resources are allocated in 

preference to others. ' ^ 

18* • CJuartile - Quarters of a population; the 25th, 50th, and 75th percentiles. 

1^ Sfchool Sentiment Index (SSI) - A collection of questions about a student's 
^ attitude toward school such, as, "Do you like 

. . ^ b-feing at school?". »The stij^dent marks the 

answer that best describesKinr. 

20* Sequential Test's of Educational Progress (STEP) - A battery of achievement 
^ ' ^ tests designed to measure 

* ^ . " student skills in the aca- 

^ / \ demic areas 'of Reading, 

^ \ * English Expressi{)n, Mechanics 

' ^ of Writing, Mathematics 

^ Computation ^nd Concepts, 

Social Studies/ and Science. 
i, ' * ^ 

21. Stattines - A ttnit of a standard score scale that divides the norm population 
info nine groups. *. ^ 



